Add combined results and plot generation by avolkov-intel · Pull Request #219 · IntelPython/scikit-learn_bench

avolkov-intel · 2026-08-17T10:58:23Z

Description

New CLI options

--combined-results — builds two extra sheets:
- All cases — all algorithm results on one page (group name, sklearn time[ms], sklearnex time[ms], speedup, then parameters). KNN is split into
  brute/kd_tree; rows sorted by dtype then dataset; each group ends with per-dtype and total GEOMEAN formula rows with a per-group speedup color scale.
- Summary (for plots) — one row per algorithm referencing the group GEOMEAN cells (total + fp32 + fp64), split into Training/Inference sections with
  section and total geomeans.
--draw-plots / --plot-output <path> — renders a Training/Inference speedup bar chart (geomean per algorithm, combining fp32 & fp64) with
Intel-branded titles and disclaimer footnote.

Other changes

--performance-stability-metrics now additionally emits a median time[ms] column (alongside existing 1st run time[ms], time CV), included in the
individual sheets and "All cases".

Use --compatibility-mode together with --combined-results so sklearn/sklearnex rows merge correctly.

Completeness and readability

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes or created a separate PR with updates and provided its number in the description, if necessary.
Git commit message contains an appropriate signed-off-by string (see CONTRIBUTING.md for details).
I have resolved any merge conflicts that might occur with the base branch.

Testing

I have run it locally and tested the changes extensively.
All CI jobs are green or I have provided justification why they aren't.
I have extended testing suite if new functionality was introduced in this PR.

david-cortes-intel · 2026-08-17T11:02:59Z

@avolkov-intel Please document these options in the .md files where the other options are described.

avolkov-intel · 2026-08-17T11:13:00Z

@avolkov-intel Please document these options in the .md files where the other options are described.

Done

avolkov-intel · 2026-08-17T13:00:11Z

CI failures are not related to this PR

david-cortes-intel · 2026-08-18T12:43:43Z

+| `--performance-stability-metrics`</br>`-psm`   |        | False                                               |                                                | Adds performance stability metrics (`1st run time[ms]`, `1st-mean run ratio`, `median time[ms]`, `time CV`) to the report.       |
+| `--combined-results`                           |        | False                                               |                                                | [EXPERIMENTAL] Creates `All cases` and `Summary (for plots)` sheets combining time and speedup of all algorithms with per-dtype/total GEOMEAN rows (use with `--compatibility-mode`). |
+| `--draw-plots`                                 |        | False                                               |                                                | [EXPERIMENTAL] Draws Training/Inference speedup bar charts from the combined results (requires `--combined-results`).             |
+| `--plot-output`                                | str    | None                                                |                                                | [EXPERIMENTAL] Output file path for plots (e.g. `plots.png`); if unset, plots are shown interactively.                           |


Does this require passing --draw-plots?

Well technically no, you can provide it without draw plots it would be just ignored

Please add that kind of detail in these docs.

david-cortes-intel · 2026-08-18T13:37:00Z

@avolkov-intel Would this work when comparing against cuml? If so, how would it handle cases where parameters do not match?

david-cortes-intel · 2026-08-18T13:38:31Z

 | `--compatibility-mode`                         |        | False                                               |                                                | [EXPERIMENTAL] Compatibility mode drops and modifies results to make them comparable (for example, sklearn and cuML parameters). |
+| `--performance-stability-metrics`</br>`-psm`   |        | False                                               |                                                | Adds performance stability metrics (`1st run time[ms]`, `1st-mean run ratio`, `median time[ms]`, `time CV`) to the report.       |
+| `--combined-results`                           |        | False                                               |                                                | [EXPERIMENTAL] Creates `All cases` and `Summary (for plots)` sheets combining time and speedup of all algorithms with per-dtype/total GEOMEAN rows (use with `--compatibility-mode`). |
+| `--draw-plots`                                 |        | False                                               |                                                | [EXPERIMENTAL] Draws Training/Inference speedup bar charts from the combined results (requires `--combined-results`).             |


This option without --plot-output doesn't appear to do anything. It doesn't show anything interactive for me, or at least not when used in an SSH session.

david-cortes-intel · 2026-08-18T13:39:51Z

@avolkov-intel Please add information about how to generate a publication-grade plot, with details in the bottom text filled in.

david-cortes-intel · 2026-08-18T13:41:08Z

+    ws = wb.create_sheet(title="All cases", index=1)
+    row, geomean_rows = 1, []
+
+    def geomean_row(label, r0, r1):


Please make row a parameter.

david-cortes-intel · 2026-08-18T13:41:59Z

+    return df
+
+
+def write_all_cases_2_sheet(dfs, wb):


Please add a description of what wb is. Better yet if you can add type hints.

david-cortes-intel · 2026-08-18T13:42:20Z

+    Each block ends with per-dtype and total GEOMEAN formula rows plus a speedup color scale.
+    Returns list of (group_name, total_geomean_row, {dtype: geomean_row}) for the summary sheet.
+    """
+    KNN = ("KNeighborsClassifier", "KNeighborsRegressor")


What about LocalOutlierFactor and NearestNeighbors?

We currently don't report performance for them, so they are not present

david-cortes-intel · 2026-08-18T13:43:22Z

+                if c in gdf.columns
+            ]
+            if sort_cols:
+                gdf = gdf.sort_values(sort_cols, kind="mergesort")


Is this trying to do a stable sort? Otherwise, why pass the parameter?

Yes, the idea is to have stable sort

david-cortes-intel · 2026-08-18T13:44:46Z

+                        j = i
+                        while j + 1 < len(dtypes) and dtypes[j + 1] == dtypes[i]:
+                            j += 1
+                        dtype_rows[dtypes[i]] = geomean_row(


Since this one doesn't embed geomean as a formula, maybe the code could be a lot more readable if this were to pre-generate subtables as dataframes and store them in a list to write at the end.

In fact the idea is indeed to embed geomean as a formula. On review they often want formulas to be present in excel spreadsheet rather than calculated outside of it for clarity. You can see here we insert it as a formula: https://github.com/IntelPython/scikit-learn_bench/pull/219/changes#diff-24c57e4b1e3d9bbd1fa3cd538fca5b437e68987456662773afc60a67afdeabc5R627

david-cortes-intel · 2026-08-18T13:45:30Z

+        return [f"={src_sheet_name}!{c}{row_num}" for c in "BCD"]
+
+    def write_section(title, rows):
+        nonlocal current_row


Please switch it to parameter.

david-cortes-intel · 2026-08-18T13:46:45Z

                new_ws.append([None])


+def draw_summary_plots(all_cases_df: pd.DataFrame, output_file: str = None):


Would be helpful to have a sub-function to generate the plot returning matplotlib objects (plt, fig, axes).

ethanglaser

LGTM if it works for you. But address David's comments to his satisfaction before merge.

avolkov-intel · 2026-09-08T13:19:42Z

@avolkov-intel Would this work when comparing against cuml? If so, how would it handle cases where parameters do not match?

no, currently it's only designed to work for sklearn vs sklearnex comparison

avolkov-intel · 2026-09-08T13:36:00Z

Addressed comments from David

avolkov-intel requested a review from david-cortes-intel as a code owner August 17, 2026 10:58

avolkov-intel force-pushed the dev/anatolyv-combined-results branch from ceda047 to 1886459 Compare August 17, 2026 11:11

ethanglaser reviewed Aug 17, 2026

View reviewed changes

Comment thread sklbench/report/compatibility.py

Comment thread sklbench/report/implementation.py

david-cortes-intel reviewed Aug 18, 2026

View reviewed changes

ethanglaser approved these changes Aug 20, 2026

View reviewed changes

avolkov-intel added 7 commits September 8, 2026 05:02

Add an option --combined-results to create list with all use cases

ca76b25

Add options to draw comparison plots

0f007d3

Handle n_jobs param correctly and add Summary 2 and All cases 2 pages

b0f520d

Add footnote for plot and add metrics

c77e14c

Update footnote

3946b64

Update docs

d1dbe1e

Code format

2341e70

Address comments

2078312

avolkov-intel force-pushed the dev/anatolyv-combined-results branch from 2f671bd to 2078312 Compare September 8, 2026 13:25

		new_ws.append([None])


		def draw_summary_plots(all_cases_df: pd.DataFrame, output_file: str = None):

Uh oh!

Conversation

avolkov-intel commented Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

New CLI options

Other changes

Uh oh!

david-cortes-intel commented Aug 17, 2026

Uh oh!

avolkov-intel commented Aug 17, 2026

Uh oh!

avolkov-intel commented Aug 17, 2026

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

avolkov-intel Aug 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-cortes-intel commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

david-cortes-intel commented Aug 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ethanglaser left a comment

Choose a reason for hiding this comment

Uh oh!

avolkov-intel commented Sep 8, 2026

Uh oh!

avolkov-intel commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

avolkov-intel commented Aug 17, 2026 •

edited

Loading

avolkov-intel Aug 18, 2026 •

edited

Loading